1. Field of the Invention
The present invention relates to a method for generating free viewpoint video image using divided local regions and more particularly to a method for generating a free viewpoint video image using images of an object photographed with a plurality of video cameras (hereinafter referred to as just camera) having each horizontal optical axis so as to surround the object.
2. Description of the Related Art
With a progress of video processing technology and video communication technology in recent years, three-dimensional free viewpoint video image has been attracting public attention as a next-generation video content. Then, a technology of generating entire circumference free viewpoint video image using multi-viewpoints photographed by disposing video cameras around an object has been studied and developed.
In this case, if cameras are disposed densely around an object in order to cope with an arbitrary viewpoint, the number of the cameras is increased thereby raising cost, which is not achievable in reality. Thus, although a method of disposing the cameras sparsely around the object is adopted, in this case, a video image located between video cameras is not obtained.
To solve this defect, conventionally, there has been proposed a method of generating a video image from a viewpoint in which an object is not photographed with the cameras by interpolating a video image between images using image based rendering.
As a typical method for the image based rendering of interpolating between multi-viewpoint images, “ray space representation method” is available, which has been described in Japanese Patent Application Laid-Open (JP-A) Nos. 2004-258775 and 10-111951 as technical documents which describe generation of interpolated video image using the ray space representation method.
However, when objects are distributed in a wide range in a real zone and parallax due to a difference in depth (depth parallax) is large, if it is intended to correct that depth parallax by the above-mentioned prior art, interpolation processing of ray information becomes complicated, which is not an actual solution.
Accordingly, the applicant has invented and filed an invention in which the real zone is divided into small local regions in which the depth parallax can be neglected and by applying the method of image based rendering to each divided local region individually, a local area free viewpoint video image is generated and then a target free viewpoint video image is generated by synthesizing those video images (Japanese Patent Application No. 2006-185648).
An outline of this invention will be described briefly with reference to FIG. 14. If it is assumed that an object video image photographed by one of plural cameras surrounding the object is (a) and a video image of a local region therein is (b), respective video images (c) in the local region photographed by the plural cameras is stored in ray space as ray information as shown in (d). Next, as shown in (e), interpolation of the ray information is carried out to charge the ray space. Next, as shown in (f), the ray information is cut out and the cut out ray information is synthesized as shown in (g) and by bring back this into the object video image region, a free viewpoint (virtual viewpoint) as shown in (h) is obtained.
According to this invention, even if objects are distributed in a wide range of real zone so that parallax due to a difference in depth is large, a video image of an arbitrary viewpoint in the real zone near the reality can be generated. Further, a video image can be generated from the virtual viewpoint between an object and other object so as to achieve walk-through experience.
However, in the above-mentioned application invention, the position of the virtual viewpoint is confined to a plane where a camera is disposed and if the virtual viewpoint is not located on that plane, for example, if the virtual viewpoint is moved upward or downward in the vertical direction with respect to the plane, the free viewpoint video image cannot be generated. In other words, if the camera is moved in the vertical direction with respect to the plane, the free viewpoint video image cannot be generated.